{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optimizing AI Trading Algorithms - Course Project\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this project you will practice optimizing various aspects of a machine learning model for predicting stock price movements. This will provide you with an opportunity to integrate the concepts covered in the course, such as data preprocessing and cleaning, hyperparameter tuning, detecting and addressing over-/under-fitting, model evaluation, and feature selection techniques. While you will use real-world data in this project, the goal is not necessarily to build a \"winning\" trading _strategy_. The goal of this course has been to equip you with the tools, techniques, concepts and insights you need to evaluate, optimize and monitor *your own* trading strategies.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Scenario"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You are an analyst at a boutique investment firm tasked with coming up with a novel idea for investing in specific sectors of the industry. You've heard that the Utilities, Consumer Staples and Healthcare sectors are relatively resilient to economic shocks and recessions, and that stock market investors tend to flock to these sectors in times of uncertainty. You decide to take the [SPDR Healthcase Sector ETF (NYSEARCA: XLV)](https://www.sectorspdrs.com/mainfund/XLV) and try to model its returns' dynamics using a machine learning AI strategy. Your novel idea is to get data for the volatility index [(INDEXCBOE: VIX)](https://www.cboe.com/tradable_products/vix/) as a proxy for uncertainty in the market. You also decide to take a look at [Google Trends data](https://trends.google.com/trends/explore?date=all&geo=US&q=recession) for the search term \"recession\" in the United States, in order to try and see if there is any meaningful relationship between the general public's level of concern about a recession happening and the price movements of the Health Care Select Sector SPDR Fund.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You decide to train a binary **classification** model that merely attempts to predict the **direction** of XLV's 5-day price movements. In other words, you want to see if on any given day, with the above data in hand, you could reliably predict whether the price of XLV will increase or decrease over the next 5 trading days."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the cell below to `import` all the Python packages and modules you will be using throughout the project."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --upgrade pip ta --quiet"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.dates as mdates\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import plotly.express as px\n",
    "import plotly.graph_objects as go\n",
    "import seaborn as sns\n",
    "import yfinance as yf\n",
    "from plotly.subplots import make_subplots\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.metrics import (\n",
    "    accuracy_score,\n",
    "    classification_report,\n",
    "    confusion_matrix,\n",
    "    f1_score,\n",
    "    precision_score,\n",
    "    recall_score,\n",
    ")\n",
    "from sklearn.model_selection import GridSearchCV, learning_curve, train_test_split\n",
    "from ta.momentum import RSIIndicator\n",
    "from ta.volatility import BollingerBands\n",
    "\n",
    "pd.options.display.max_columns = 50\n",
    "pd.options.display.max_rows = 50\n",
    "\n",
    "RANDOM_SEED = 42"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Data Acquisition, Exploration, Cleaning and Preprocessing\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, you will download and inspect:\n",
    "\n",
    "- daily data for the SPDR Healthcase Sector ETF (NYSEARCA: XLV)\n",
    "- daily data for the volatility index (INDEXCBOE: VIX)\n",
    "- monthly data from Google Trends for the search interest in the term \"recession\" in the United States\n",
    "\n",
    "The goal is to make sure the data is clean, meaningful, and usable for selecting and engineering features.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1. Price and Volume Data for \"XLV\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have downloded daily data from **January 1st, 2004** to **March 31st, 2024** for the ticker **XLV** using the `yfinance` library and stored it in a CSV file named `xlv_data.csv`. Load this data into a Pandas DataFrame named `xlv_data`, making sure to set the index column to the first column of the CSV file (`Date`) and set `parse_dates=True`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xlv_data = ## YOUR CODE HERE ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the `info()` and `describe()` methods to get an overview of how many rows of data there are in `xlv_data`, what columns are present and what their data types are, and what some basic statistics (mean, std, quartiles, min/max values) of the columns look like.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## YOUR CODE HERE ##"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## YOUR CODE HERE ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How many `NaN` rows are there in `xlv_data`?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "answer = ## YOUR CODE HERE ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a look at the final five rows of `xlv_data`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "xlv_data.tail()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Raw OHLC data is not suitable for training models. The absolute price level of a security is boundless in theory and not particularly menaningful. In the next section, you are going to engineer useful features from all of these columns. For now, as a visual sanity check, plot `Adj Close` as a line plot.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## YOUR CODE HERE ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Bonus**: The cell below plots the combined candlestick + volume chart for the last 15 months of data using Plotly.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_since_2023 = xlv_data[\"2023-01-01\":]\n",
    "\n",
    "figure = make_subplots(specs=[[{\"secondary_y\": True}]])\n",
    "figure.add_traces(\n",
    "    go.Candlestick(\n",
    "        x=data_since_2023.index,\n",
    "        open=data_since_2023.Open,\n",
    "        high=data_since_2023.High,\n",
    "        low=data_since_2023.Low,\n",
    "        close=data_since_2023.Close,\n",
    "    ),\n",
    "    secondary_ys=[True],\n",
    ")\n",
    "figure.add_traces(\n",
    "    go.Bar(x=data_since_2023.index, y=data_since_2023.Volume, opacity=0.5),\n",
    "    secondary_ys=[False],\n",
    ")\n",
    "\n",
    "figure.update_layout(\n",
    "    title=\"XLV Candlestick Chart Since 2023\",\n",
    "    xaxis_title=\"Date\",\n",
    "    yaxis_title=\"Volume\",\n",
    "    yaxis2_title=\"Price\",\n",
    "    showlegend=False,\n",
    ")\n",
    "figure.update_yaxes(fixedrange=False)\n",
    "figure.layout.yaxis2.showgrid = False\n",
    "figure.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2. Data for The Volatility Index `VIX`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As before, we have downloaded daily data for the volatility index [(INDEXCBOE: VIX)](https://www.cboe.com/tradable_products/vix/) over the same time period using `yfinance` and provided it to you in a CSV file named `vix_data.csv`. Load the data into a variable named `vix_data`. Make sure to set the index and parse the dates correctly.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vix_data = ## YOUR CODE HERE ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot a line chart of the `Adj Close` value of the VIX using your method of choice (e.g. `plotly` or `matplotlib`).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## YOUR CODE HERE ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3. Google Trends Data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **monthly** evolution of search interest in the term \"recession\" in the U.S. over the period of interest (Jan. 2003 - Mar. 2024) from the Google Trends website has been provided to you as a CSV file. We will load this data using Pandas into a DataFrame named `google_trends_data`, set the index column of the DataFrame to the \"`Month`\" column from the CSV and have Pandas try and parse these dates automatically.\n",
    "\n",
    "> Note: The \"Month\" column in the CSV is in \"YYYY-MM\" format.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "google_trends_data = ## YOUR CODE HERE ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As noted above, the CSV lists **monthly** search trends data and the `Month` column is in YYYY-MM format. How has Pandas interpreted and parsed these into specific dates? Take a look at `google_trends_data`'s index.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "google_trends_data.index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We would have liked to assign the data points to the last day of the respective months, as this data would have been available at the _end_ of each period. Shift the index column of `google_trends_data` to do this.\n",
    "\n",
    "> Hint: You can use `pd.offsets.MonthEnd()` from Pandas.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "google_trends_data.index = ## YOUR CODE HERE ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the cell below to visualize this data as a line plot.\n",
    "\n",
    "> **Note from Google:** \"Numbers represent search interest relative to the highest point on the chart for the given region and time. A value of 100 is the peak popularity for the term. A value of 50 means that the term is half as popular. A score of 0 means there was not enough data for this term.\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(12, 6))\n",
    "ax.plot(google_trends_data)\n",
    "date_fmt = mdates.DateFormatter(\"%Y-%m\")\n",
    "plt.xlim(google_trends_data.index[0], google_trends_data.index[-1])\n",
    "plt.xticks(rotation=45)\n",
    "plt.title(\"Google Trends Data (Monthly)\")\n",
    "plt.legend([\"Search interest over time in the term 'recession' in the US\"])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But not every month-end is a trading day. Also, what value should the model train on for all the days in between month-ends? Below, we have provided you with code to convert the monthly data to daily and interpolate the end-of-month values to get all the in-between values. You will be using this new `google_trends_daily` data going forward.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "ruby"
    }
   },
   "outputs": [],
   "source": [
    "google_trends_daily = google_trends_data.resample('D').asfreq().interpolate(method='linear')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "ruby"
    }
   },
   "outputs": [],
   "source": [
    "# # The shape of the chart should not have changed\n",
    "# google_trends_daily.plot.line(title=\"Google Trends Data (Daily)\", figsize=(12, 6)).legend(\n",
    "#     labels=['Search interest over time in the term \"recession\" in the US']\n",
    "# );"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Feature Engineering and Analysis\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, you will create a new DataFrame called `data` which will house all of the features as well as the prediction target. Then you will analyze the features and look for potentially problematic features.\n",
    "\n",
    "Start by running the cell below to create `data` as an empty DataFrame with just an index that matches `XLV`'s.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = ## YOUR CODE HERE ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1. Feature Engineering\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.1.1. Month and Weekday\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add the `month` and `weekday` columns to `data` as categorical features (integer labels) from its index.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## YOUR CODE HERE ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You do not want to train a model using these columns as they are, because the numbers themselves and the inherent \"order\" of months and weekdays do not really have any significance, but the model may interpret them as meaningful. You could either (a) use one-hot encoding to turn each category to a separate binary feature, or (b) treat them as cyclical features. The choice is somewhat arbitrary and depends on how important a \"feature\" you believe the cyclicality to be.\n",
    "\n",
    "Below, you will:\n",
    "\n",
    "- Treat `month` as a cyclical feature, creating two features (`month_sin` and `month_cos`). (👉 See: [Trigonometric features](https://scikit-learn.org/stable/auto_examples/applications/plot_cyclical_feature_engineering.html#trigonometric-features))\n",
    "- One-hot-encode `weekday` and create five additional features of type `int32` (one for each business day) with the `weekday` prefix. (👉 See: [`pandas.get_dummies()`](https://pandas.pydata.org/docs/reference/api/pandas.get_dummies.html))\n",
    "- Make sure the original `month` and `weekday` columns are no longer present in `data`. (`drop()` them if necessary.)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Treat `month` as a \"cyclical\" feature with a period of 12 months.\n",
    "data[\"month_sin\"] = ## YOUR CODE HERE ##\n",
    "data[\"month_cos\"] = ## YOUR CODE HERE ##\n",
    "\n",
    "# Drop the original `month` column.\n",
    "## YOUR CODE HERE ##\n",
    "\n",
    "# Treat `weekday` as a \"categorical\" feature and one-hot-encode it.\n",
    "data = pd.get_dummies(## YOUR CODE HERE ##)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.1.2. Historical Returns\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, add features for historical returns of the XLV ETF from its `Adj Close` column. For each date, calculate rolling **simple** returns over the past 1, 5, 10 and 20 days. Create 4 columns in `data` named `ret_#d_hist` where `#` is the lookback period. The list `hist_ret_lookbacks` is provided if you wish to use it.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create features for 1-day, 5-day, 10-day and 20-day historical returns\n",
    "hist_ret_lookbacks = [1, 5, 10, 20]\n",
    "## YOUR CODE HERE ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The cell below plots the histograms of the returns you just calculated. They should look normally distributed around zero.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hist_ret_lookbacks = [1, 5, 10, 20] # In case it was deleted from the previous cell\n",
    "\n",
    "fig, axs = plt.subplots(2, 2, figsize=(10, 8))\n",
    "\n",
    "\n",
    "def plot_hist_returns(ax, data, col, title):\n",
    "    ax.hist(data[col], bins=200)\n",
    "    ax.set_title(title)\n",
    "\n",
    "\n",
    "for i, n_days in enumerate(hist_ret_lookbacks):\n",
    "    plot_hist_returns(\n",
    "        axs[i // 2, i % 2], data, f\"ret_{n_days}d_hist\", f\"Distribution of Historical {n_days}-Day Returns\"\n",
    "    )\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.1.3. Trade Volumes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As trading volumes span several orders of magnitude, take the natural logarithm of `Volume` and use it as a feature instead. This helps emphasize variations in its lower range. Use `np.log()` and call this new feature `log_volume`.\n",
    "\n",
    "**Note:** For tree-based models such as Decision Trees and Random Forests, scaling is not necessary. But feature scaling becomes critically important if you use other model types (e.g. distance-based models).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data[\"log_volume\"] = ## YOUR CODE HERE ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.1.4. Technical Indicators"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add a feature named `ibs` which is calculated as (Close - Low) / (High - Low). This measure, a number between zero and one and sometimes referred to as the \"Internal Bar Strength\", denotes how \"strong\" the closing price is relative to the high and low prices within the same period.\n",
    "> **Note:** Make sure to use `Close` (not `Adj Close`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Engineer the technical indicator \"Internal Bar Strength\" (IBS) from XLV's price data\n",
    "data[\"ibs\"] = ## YOUR CODE HERE ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the cell below to add a few more technical indicators, including Bollinger Band features and indicators, as well as the Relative Strength Index (RSI)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get some more technical indicators using the `ta` library\n",
    "\n",
    "indicator_bb = BollingerBands(close=xlv_data[\"Close\"], window=20, window_dev=2)\n",
    "indicator_rsi = RSIIndicator(close=xlv_data[\"Close\"], window=14)\n",
    "\n",
    "# Add Bollinger Bands features\n",
    "data[\"bb_bbm\"] = indicator_bb.bollinger_mavg()\n",
    "data[\"bb_bbh\"] = indicator_bb.bollinger_hband()\n",
    "data[\"bb_bbl\"] = indicator_bb.bollinger_lband()\n",
    "\n",
    "# Add Bollinger Band high and low indicators\n",
    "data[\"bb_bbhi\"] = indicator_bb.bollinger_hband_indicator()\n",
    "data[\"bb_bbli\"] = indicator_bb.bollinger_lband_indicator()\n",
    "\n",
    "# Add Width Size and Percentage Bollinger Bands\n",
    "data[\"bb_bbw\"] = indicator_bb.bollinger_wband()\n",
    "data[\"bb_bbp\"] = indicator_bb.bollinger_pband()\n",
    "\n",
    "# Add RSI\n",
    "data[\"rsi\"] = indicator_rsi.rsi()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.1.5. The Target of Prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add the column `tgt_is_pos_ret_5d_fut` as type `int` to `data`, denoting whether forward-looking 5-day returns on each day are positive (a value of `1`) or negative (a value of `0`).\n",
    "> **Note:** Again, as before, calculte **simple** returns from the `Adj Close` column of `xlv_data`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the prediction target: an integer indicating whether future 5-day returns are positive (1) or negative (0)\n",
    "## YOUR CODE HERE ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the cells below to get an idea of how balanced the distribution of the target variable is throughout the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_col_name = \"tgt_is_pos_ret_5d_fut\"\n",
    "# Inspect the distribution of the target variable\n",
    "target_value_counts = data[target_col_name].value_counts()\n",
    "target_value_counts / len(data)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_value_percentages = target_value_counts / len(data) * 100\n",
    "\n",
    "plt.bar(target_value_percentages.index.astype(str), target_value_percentages.values)\n",
    "plt.xlabel(\"Target Variable: Positive 5-day Forward-Looking Return (1=Yes, 0=No)\")\n",
    "plt.ylabel(\"Percentage of Observations (%)\")\n",
    "plt.title(\"Distribution of Target Variable\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Does the data look relatively balanced or grossly unbalanced in the distribution of the target variable? Why is this important?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "answer = \"## YOUR ANSWER HERE ##\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.1.6. Stitching Everything Together"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will now add the `vix_data` and `google_trends_daily` as features to `data`. You will also rename the column corresponding to the VIX feature. Run the cell below to do so."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Join with the Google Trends data and VIX data\n",
    "data = data.join(google_trends_daily, how=\"left\")\n",
    "data = data.join(vix_data[\"Adj Close\"], how=\"left\")\n",
    "data.rename(columns={\"Adj Close\": \"vix\"}, inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2. Further Data Preprocessing and Cleaning\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While engineering new features, some `NaN` values were created. You now need to clean the combined DataFrame. Inspect `data` to see how many `NaN` values there are per column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## YOUR CODE HERE ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some features, such as historical returns, RSI, Bollinger Bands and BB indicators cannot be calculated for the first `n` days due to their \"rolling\" nature. In general, missing values can sometimes be imputed with reasonable estimates. But here you will simply drop the rows containing them. The largest `n` is `20`, corresponding to the calculation of 20-day historical returns. Drop the first 20 rows of `data`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## YOUR CODE HERE ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Are there any more missing values?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## YOUR CODE HERE ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Even if there aren't, you remember that when you calculated the target variable (`tgt_is_pos_ret_5d_fut`) based on forward-looking 5-day rolling returns, you could not have known future returns for the last five days of `data`! Therefore the last 5 rows of data should be dropped.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## YOUR CODE HERE ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us take a final look at the types and statistical characteristics of the set of features and targets.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3. Correlation Analysis\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Correlation analysis can be a rough and early form of feature importance analysis. Features that are highly correlated (in either direction) with each other but not with the target variable, are a sign of multicollinearity problems, which means they may not contribute much additional information in predicting the target. In fact, depending on the algorithm used, multicollinearity may result in stability and reliability issues. Checking the correlation matrix can be helpful in identifying such features.  \n",
    "\n",
    "Plot the heatmap of the correlation matrix of features/target and identify a cluster of 3 features that are almost certainly collinear. (Hint: `bb_bbm` is one of them.) You can pass the correlation matrix directly to [Seaborn's `heatmap()` method](https://seaborn.pydata.org/generated/seaborn.heatmap.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## YOUR CODE HERE ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In such scenarios, we usually eliminate all but one of the collinear features. Keep `bb_bbm` and drop the other two features that are highly linearly related to it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## YOUR CODE HERE ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is also one feature that is very highly correlated with `rsi` (which makes intuitive sense, as it, too, is a measure of relative strength). Find it an eliminate it, leaving `rsi` intact."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "## YOUR CODE HERE ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the heatmap of the new, reduced correlation matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## YOUR CODE HERE ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Features that are highly correlated (negatively or positively) **with the target variable** are likely more important. Which two (2) independent variables (features) are correlated more than 4% (**in either direction**) with the boolean target variable denoting whether 5-day future returns are positive?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## YOUR CODE HERE ##\n",
    "answer = \"## YOUR ANSWER HERE ##\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. The Training-Validation-Testing Split\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, you will split the `data` set into two sets: the training and validation set, and the testing set. You will then come up with a baseline score so that you have a reference point for evaluating your model's performance.\n",
    "\n",
    "**Note:** Technically, since you are not going to use classical statistics-based time-series prediction methods (such as ARIMA), you can shuffle the data before splitting it. But for ease of interpretability and backtesting, you may as well keep the data in its original order. This is fine as long as the distributions of features and the target variable do not significantly shift over time. - And that is an important assumption related to drift analysis, which was covered in the course, but we will not get to in this project."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1. The Split\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is time to split the data, temporally, into the training + validation and testing sets. You will train and optimize (i.e. cross-validate) your model on the first 80% of the data, and use the remaining 20% for the test set (i.e. to evaluate the performance of your model). Use the `train_test_split()` method from scikit-learn's `model_selection` module to perform the split.\n",
    "> **Note:** Please make sure to set `shuffle=False` and `random_state=RANDOM_STATE`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## YOUR CODE HERE ##\n",
    "\n",
    "X_train_val, X_test, y_train_val, y_test = ## YOUR CODE HERE ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2. Baseline Model and Score\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Earlier, you inspected the distribution of the target variable across the entire data set. Run the cell below to analyze at the distribution of the target variable in each split."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_val_pct = y_train_val.value_counts(normalize=True) * 100\n",
    "test_pct = y_test.value_counts(normalize=True) * 100\n",
    "\n",
    "categories = [\"Train + Validation\", \"Test\"]\n",
    "zero_counts = [train_val_pct[0], test_pct[0]]\n",
    "one_counts = [train_val_pct[1], test_pct[1]]\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(10, 6))\n",
    "\n",
    "ax.bar(categories, zero_counts, label=\"0\")\n",
    "ax.bar(categories, one_counts, bottom=zero_counts, label=\"1\")\n",
    "\n",
    "# Add text annotations\n",
    "for i, (zero, one) in enumerate(zip(zero_counts, one_counts)):\n",
    "    ax.text(i, zero / 2, f\"{zero:.2f}%\", ha=\"center\", va=\"center\", color=\"white\")\n",
    "    ax.text(\n",
    "        i,\n",
    "        zero + one / 2,\n",
    "        f\"{one:.2f}%\",\n",
    "        ha=\"center\",\n",
    "        va=\"center\",\n",
    "        color=\"white\",\n",
    "    )\n",
    "ax.set_title(f\"Distribution of {target_col_name} in the Train/Validation vs. Test Set for XLV\")\n",
    "ax.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you were to devise a simple model that naively always predicted the majority class, what would the accuracy score of your model be on the training+validation set? How about on the testing set? Consider the latter your baseline score, i.e. a reference score to compare your more sophisticated model's performace to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "baseline_accuracy_test_score = ## YOUR CODE HERE ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Model Training and Tuning\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, you will train a `RandomForestClassifier`, a robust, versatile ensemble learning method that uses \"bagging\" (also known as \"bootstrap aggregating\") to train multiple Decision Trees. The technical details of the model are beyond the scope of this course, but you may read more about it [here](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the cell below which defines a function that allows you to plot learning curves annotated with a hyperparameter named `max_depth` which you pass to it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_learning_curves(train_sizes, train_scores, test_scores, max_depth, axs):\n",
    "    train_scores_mean = np.mean(train_scores, axis=1)\n",
    "    train_scores_std = np.std(train_scores, axis=1)\n",
    "    test_scores_mean = np.mean(test_scores, axis=1)\n",
    "    test_scores_std = np.std(test_scores, axis=1)\n",
    "\n",
    "    axs.fill_between(\n",
    "        train_sizes,\n",
    "        train_scores_mean - train_scores_std,\n",
    "        train_scores_mean + train_scores_std,\n",
    "        alpha=0.1,\n",
    "        color=\"b\",\n",
    "    )\n",
    "    axs.fill_between(\n",
    "        train_sizes,\n",
    "        test_scores_mean - test_scores_std,\n",
    "        test_scores_mean + test_scores_std,\n",
    "        alpha=0.1,\n",
    "        color=\"r\",\n",
    "    )\n",
    "    axs.plot(\n",
    "        train_sizes,\n",
    "        train_scores_mean,\n",
    "        \"o-\",\n",
    "        color=\"b\",\n",
    "        label=\"Average Score on Training Sets\",\n",
    "    )\n",
    "    axs.plot(\n",
    "        train_sizes,\n",
    "        test_scores_mean,\n",
    "        \"o-\",\n",
    "        color=\"r\",\n",
    "        label=\"Average Score on Test Sets\",\n",
    "    )\n",
    "    axs.set_xlabel(\"Training examples\")\n",
    "    axs.set_ylabel(\"Score\")\n",
    "    axs.set_title(f\"Learning Curves (max_depth={max_depth})\")\n",
    "    axs.legend(loc=\"center left\")\n",
    "    axs.grid(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below is the first iteration of your model. It uses the default values for most of its hyperparameters. We have only specified one hyperparameter, `max_depth=10`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "max_depth = 10\n",
    "model = RandomForestClassifier(max_depth=max_depth, random_state=RANDOM_SEED, n_jobs=-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the `learning_curve()` method from scikit-learn's `model_selection` module to cross-validate your model, with `accuracy` as the `scoring` metric. Use 10%, 20%, 30%,... , and 100% of the training+validatin data, with 5-fold cross-validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## YOUR CODE HERE ##\n",
    "\n",
    "train_sizes, train_scores, test_scores = ## YOUR CODE HERE ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inspect the learning curves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "figure = plt.figure(figsize=(10, 6))\n",
    "axs = figure.gca()\n",
    "\n",
    "plot_learning_curves(train_sizes, train_scores, test_scores, max_depth, axs)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wondering what effect different values of the `max_depth` hyperparameter have, you decide to experiment with lower (`10`) and higher (`20`) values of it to see how the plots change. Run the cell below to help you answer the questions that follow it.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axs = plt.subplots(1, 3, figsize=(14, 6))\n",
    "max_depth_range = [5, 10, 15]\n",
    "for i, max_depth in enumerate(max_depth_range):\n",
    "    model = RandomForestClassifier(max_depth=max_depth, random_state=RANDOM_SEED, n_jobs=-1)\n",
    "    train_sizes, train_scores, test_scores = learning_curve(\n",
    "        model, X_train_val, y_train_val, train_sizes=train_sizes, cv=5, scoring=\"accuracy\"\n",
    "    )\n",
    "    plot_learning_curves(train_sizes, train_scores, test_scores, max_depth, axs[i])\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With a value of `max_depth=15`, does your model overfit or underfit?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "answer = \"## YOUR ANSWER HERE ##\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With a value of `max_depth=15`, is your performance metric (accuracy score) more likely to improve with more training data or with higher model complexity?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "answer = \"## YOUR ANSWER HERE ##\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Random Forest Classifiers have several other hyperparameters, such as `min_samples_split` (default=2), `min_samples_leaf` (default=1) and `n_estimators` (default=100). So far, you have been tuning your model manually. But with all the possible combinations of hyperparameters, this is not tractable.  \n",
    "\n",
    "Use grid search cross-validation (the `GridSearchCV` class from scikit-learn's `model_selection` module) to find the optimal combination of hyperparameters from the search space specified below:\n",
    "- `max_depth` = 2, 3, 4 or 5\n",
    "- `min_samples_leaf` = 1, 2, 3 or 4\n",
    "- `n_estimators` = 50, 75, 100, 125, or 150\n",
    "\n",
    "As before, use 5-fold cross-validation and accuracy as the `scoring` metric. Name your tuning model `search`.\n",
    "\n",
    "> Note: Setting `n_jobs=-1` will allow Python to take advantage of parallel computing on your computer to speed up the training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grid = ## YOUR CODE HERE ##\n",
    "\n",
    "search = ## YOUR CODE HERE ##\n",
    "\n",
    "search.fit(## YOUR CODE HERE ##)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the cell below to see the top 5 best performing hyperparameter combinations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.DataFrame(search.cv_results_).sort_values(\"rank_test_score\").head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at the results of GridSearchCV, which hyperparameters yield the highest mean test score?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_max_depth = ## YOUR CODE HERE ##\n",
    "best_min_samples_leaf = ## YOUR CODE HERE ##\n",
    "best_n_estimators = ## YOUR CODE HERE ##\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking more closely at the DataFrame of top 5 results, varying which hyperparameter did not seem to have any effect, at least in the top-ranking score?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "answer = \"## YOUR ANSWER HERE ##\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Model Evaluation and Interpretation\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, you will evaluate the performance metrics of the best model you found in the previous section and analyze feature importance in relation to model performance."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.1. Evaluation (Performance Metrics)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is finally time to train your model on the entire training + validation set with the optimal set of hyperparameters you just found, and evaluate its performance on the test set.  \n",
    "\n",
    "Train (`fit()`) a `RandomForestClassifier` on the training data with the optimal combination of hyperparameters you found in the previous section. Name it 'clf'.\n",
    "\n",
    "> **Note:** Remember to set `random_state=RANDOM_SEED` for consistency of results, and set `n_jobs=-1` to automatically speed up the run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = ## YOUR CODE HERE ##\n",
    "clf.fit(## YOUR CODE HERE ##)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Store your trained model's predictions on the **testing** set in a variable named `y_test_pred`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_test_pred = ## YOUR CODE HERE ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Complete the Python dictionary in the code cell below to evaluate your model and answer the questions that follow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "evaluation = {\n",
    "    \"accuracy\": ## YOUR CODE HERE ##,\n",
    "    \"precision\": ## YOUR CODE HERE ##,\n",
    "    \"recall\": ## YOUR CODE HERE ##,\n",
    "    \"f1\": ## YOUR CODE HERE ##,\n",
    "}\n",
    "display(evaluation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Explain, in words and citing the actual numbers from the evaluation report above, what the **precision** and **recall** scores mean."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "answer = \"## YOUR ANSWER HERE ##\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the cell below to get a more detailed report."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(classification_report(y_test, y_test_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How many True Negatives, False Negatives, False Positives and True Positives did the model predict on the test set? Find out using the `confusion_matrix()` method from scikit-learn's `metrics` module. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "confusion_matrix(y_test, y_test_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Answer the question from earlier.\n",
    "> Note: Feel free to rename the variables. We will not reference them later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_TrueNeg = ## YOUR CODE HERE ##\n",
    "num_FalseNeg = ## YOUR CODE HERE ##\n",
    "num_FalsePos = ## YOUR CODE HERE ##\n",
    "num_TruePos = ## YOUR CODE HERE ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Is the model overfitting or underfitting? Did it manage to capture the variance on the training set but fail to generalize to the testing set? Take a look at the `classification_report()` and `confusion_matrix()` on the **training** data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train_val_pred = ## YOUR CODE HERE ##\n",
    "print(classification_report(y_train_val, y_train_val_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "confusion_matrix(y_train_val, y_train_val_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How does your model's performance compare to the baseline in terms of `accuracy`?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "answer = \"## YOUR ANSWER HERE ##\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How do the `precision` and `recall` of your model compare to those of the baseline model?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "answer = \"## YOUR ANSWER HERE ##\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2. Revisiting Feature Importance\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You decide to see if there are any features that are not contributing significantly to the performance of the model. Use the `feature_importances_` property of your classifier.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## YOUR CODE HERE ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a new training set named `X_train_val_reduced` and a new testing set named `X_test_reduced` by eliminating any features from the old train/test sets that had a feature importance of less than `0.5%`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop features that have an importance of 0.05% or less...\n",
    "feats_to_drop = ## YOUR CODE HERE ##\n",
    "X_train_val_reduced = ## YOUR CODE HERE ##\n",
    "X_test_reduced = ## YOUR CODE HERE ##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Re-do your grid search cross-validation with the same grid of hyperparameters as before but with the **reduced** feature set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = RandomForestClassifier(random_state=RANDOM_SEED, n_jobs=-1)\n",
    "\n",
    "search = ## YOUR CODE HERE ##\n",
    "\n",
    "search.fit(## YOUR CODE HERE ##)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.DataFrame(search.cv_results_).sort_values(\"rank_test_score\").head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "search.best_params_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train a new classifier on the reduced feature set with the best hyperparameters combination from the new grid search and then inspect its accuracy on the test set (with the **reduced** feature set)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clf = ## YOUR CODE HERE ##\n",
    "clf.fit(## YOUR CODE HERE ##)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_test_pred = ## YOUR CODE HERE ##\n",
    "evaluation = {\n",
    "    \"accuracy\": ## YOUR CODE HERE ##,\n",
    "    \"precision\": ## YOUR CODE HERE ##,\n",
    "    \"recall\": ## YOUR CODE HERE ##,\n",
    "    \"f1\": ## YOUR CODE HERE ##,\n",
    "}\n",
    "display(evaluation)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(classification_report(y_test, y_test_pred, zero_division=0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "confusion_matrix(y_test, y_test_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How does the accuracy compare to your last trained model?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "answer = \"## YOUR ANSWER HERE ##\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How does the accuracy compare to the baseline?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "answer = \"## YOUR ANSWER HERE ##\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a look at the classification report and confusion matrices on the **training data** with the **reduced feature set** as well:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train_val_pred = ## YOUR CODE HERE ##\n",
    "print(classification_report(y_train_val, y_train_val_pred, zero_division=0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "confusion_matrix(y_train_val, y_train_val_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What would your next course of action be? In particular, share your thoughts on the following:\n",
    "- Further optimization of this model\n",
    "- Pursuing a different trading strategy or market (instruments) altogether\n",
    "- Anything else?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "answer = \"## YOUR ANSWER HERE ##\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do you think of the fact that we used interpolated **monthly** Google Trends data to try and predict short-term (5-day) price movements? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "answer = \"## YOUR ANSWER HERE ##\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These results highlight the challenges in consistently training AI/ML models that outperform naive baseline scores in financial markets due to factors such as non-stationary data, low signal-to-noise ratio, high market efficiency, and a competitive and adversarial trading environment. It would be necessary to gather much more data (and higher quality data) than we have in this project, and to engineer much more complex features and models to eke out even a slight gain in performance. It is therefore essential to use your domain knowledge, have realistic expectations, and constantly monitor your modeling assumptions and metrics. We hope that this project enables you to do so by giving you the tools, techniques and ideas to keep in mind."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "udacity",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
